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Abstract 



We consider a model in which each family transforms under a 

different SU(3) color group. The low-energy effective theory is an 

extension of the Standard Model, with additional color octet gauge 

I bosons Gh with mass M that couple preferentially to the third family 

' quarks. We show that there are two distinct regions of the model's 

I parameter space in which we can simultaneously evade all the current 

I experimental constraints, one with M ~ 250 GeV and the other with 

' M > 600 GeV. Within each allowed region, we can obtain a correction 

I ' to the Zbb vertex that is consistent with the slightly high value of Rf, 

Qh! 

Q I observed at LEP. We show that there are AB = 1 operators in our 

'■^ model that can suppress the 5-meson semileptonic branching ratio 

BsL and the charm multiplicity per decay Uc by enough to reconcile 
the spectator parton model predictions with the experimental data. 
5^ I In the non-supersymmetric version of our model, we can only obtain 

the desired corrections to Rb, Bsl and Uc in different regions of the 
allowed parameter space, while in the supersymmetric version, we can 
obtain all three corrections simultaneously. We also discuss a strong- 
coupling limit of our model in which the third-family quarks become 
composite. 
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1 Introduction 



Despite its enormous success, the Standard Model gives us no explanation 
of the origin of flavor. The reason for the existence of three generations of 
fermions and for their hierarchical pattern of masses and mixing angles is, at 
present, not understood. The flavor physics of the third generation has been 
particularly mysterious. The smallness of the third generation mixing angles, 
and the huge hierarchy between the top and bottom quark masses have lead 
some to speculate that the third generation may be fundamentally different 
from the other two. Recent experimental results have also helped to moti- 
vate such speculation. There are a number of small discrepancies between 
the current experimental data and the expectations of the Standard Model 
concerning properties of the 6-quark. The observed 5-meson semileptonic 
branching ratio is much lower than the spectator parton model prediction, 
and the charm multiplicity in the decay products is also low, in both cases by 
roughly 20% Another possible anomaly is the high value of the Z ^ bb 
width observed at LEP, though in this case the discrepancy with the Stan- 
dard Model prediction is much smaller. Whether or not these are signs of 
new physics or merely systematic errors is subject to debate 0, Never- 
theless, it is natural to question whether there are models that can explain 
the observed pattern of discrepancies, while remaining consistent with all the 
other relevant experimental constraints. 

In this paper, we consider a model based on the gauge group SU(3)iX 
SU(3)2X SU(3)3xSU(2)LxU(l)y. Quarks belonging to generation x trans- 
form as triplets under SU(3)2,., but as singlets under the other two SU(3)'s. 
We assume that SU(3)ixSU(3)2 breaks to its diagonal subgroup SU(3)i^2 at 
some high scale A^, and that SU(3)i^2X SU(3)3 breaks to its diagonal sub- 
group, ordinary QCD, at a lower scale A^. As we will see, the effective theory 
below Ai is simply the Standard Model with additional massive color octet 
gauge bosons, Gh, that couple preferentially to the third generation quarks. 
We will show that there are two regions of the parameter space of the model 
that are consistent with the constraints from the top quark production cross 
section, from collider searches for new particles decaying to dijets, and from 
flavor changing neutral current (FCNC) processes. In the latter case, we use 
the approach of Froggatt and Nielsen ^ to construct mass matrices that lead 
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to a natural suppression of the neutral flavor changing qGuq couplings that 
arise at tree-level in the model. We show that it is possible within both of 
these allowed regions to obtain corrections to the nonleptonic -B-decay widths 
that can reconcile the spectator parton model predictions for the semileptonic 
branching ratio and the average charm multiplicity per decay with the mea- 
sured values. We also can obtain a correction to the Z ^ bb width that 
reconciles the Standard Model expectation with the value observed at LEP, 
but in a different region of the parameter space. In the supersymmetric ver- 
sion of our model, however, we can obtain all three effects simultaneously. 
As a consequence of the new contribution to the Z ^ bb width, the value 
of as{mz) extracted from r(Z — > hadrons) is reduced. This shift is suffi- 
cient to reconcile the LEP value of ag with the measurements made at lower 
energies 0|. 

We should point out that the idea of introducing different gauge groups 
for each generation has appeared in various forms throughout the litera- 
ture. However, most of the early references do not focus on the detailed 
phenomenology of the models [^], while the more recent references consider 
different SU(2)iS rather than SU(3)s to explain a now nonexistent anomaly 
in T decay [^]. Our work is also similar in some respects to the recent litera- 
ture on the phenomenology of topcolor models, which also involve additional 
massive color octet bosons 0. In contrast to these references, we focus on 
the problem of constructing realistic mass matrices and the associated flavor 
physics. Since it is our purpose here to explore the anomalies described above, 
rather than to explain electroweak symmetry breaking through top condensa- 
tion, we do not restrict ourselves to the case where the qGuq coupling attains 
its critical value. In the strong-coupling limit of our model, we consider the 
possibility that the third generation quarks may become composite, in a way 
similar to the Abbott-Farhi model [Q, without any condensation at all. 

The paper is organized as follows. In Section ||, we present both the 
non-supersymmetric and supersymmetric versions of our model, and discuss 
the specific structure of the fermion mass matrices. The latter is crucial in 
determining the constraints from flavor-changing processes that we present 
in Sections ^ and |^. In Section |^ we identify the allowed parameter space 
of our model, and show that we can reproduce the value of Rh measured at 
LEP within the two allowed regions. In Section ^, we consider the AB = 1 
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and AB = 2 operators that result from a slightly more complicated form of 
the down-quark mass matrix. We show that the new AB = 1 operators can 
reduce both the 5-meson semileptonic branching ratio Bsl and the average 
charm multiplicity per decay ric enough to reconcile the experimental data 
with the spectator parton model predictions. We show that the strong con- 
straints from 6 ^ S7 in the non-supersymmetric version of the model only 
allow us to account for Rh and the B decay anomalies in different regions of 
the allowed parameter space. However, in the supersymmetric version, the 
b ^ S'-y branching fraction is suppressed, allowing us to explain Rb, Bsl and 
ric simultaneously. In Section ^ we consider what may happen in the con- 
fining phase of the color SU(3) corresponding to the third family, and argue 
that the bottom and top quarks may become composite. In the final section 
we summarize our conclusions. 



2 The Model 

We assume that the color gauge group SU(3)i xSU(3)2xSU(3)3 is broken 
to SU(3)i_2xSU(3)3 at some high scale Ah > (9(100) TeV by the vacuum 
expectation value (vev) of a Higgs boson $1^2 transforming as a (3,3*,1). In 
the effective theory below Ah, the first and second generation fields have 
color charges under SU(3)i_2 and the third generation fields under SU(3)3. 
The group SU(3)i^2xSU(3)3 is broken to its diagonal subgroup, SU(3)c, at 
a much lower scale by the vev of a Higgs boson $2,3 transforming as 
(1,3,3*) under the original color group. The QCD coupling gs is related to 
the couplings gi^2 and by 

1 _ 1 1 

9 2 ~^ 2' \J 

9s 91,2 93 

and thus we use the parameterization 

91,2 = (2) 
cos 6' 

93 = ^n- (3) 

sm 6 

The last stage of symmetry breaking leaves a color-octet vector boson Gh 
with mass M ~ gsAL/{sm9 cos9) and with the following coupling to the 
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quarks: 
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+ down-sector 



(4) 

Here, the arc SU{3) generators and the primed quark fields denote in- 
teraction cigcnstates, which differ from the mass cigenstates in generaL The 
symmetry breaking also leaves a color-octet scalar boson in the low-energy 
theory. The Higgs field $2,3 contains two color-octets under the diagonal 
SU(3)c; one of them is eaten by the vector boson Gh, while the other re- 
mains as a physical scalar multiplet. 

The hierarchy of scales in the model is rendered natural through super- 
symmetry. The model can be easily supcrsymmctrizcd by introducing Higgs 
chiral superfields $2,3 transforming as (1,3,3*) and $23 (1)3*, 3). The 
most general superpotential below the scale Ah is 



W = //Tr$$' + Met$ + h'det^', 



(5) 



with a mass parameter fi, and coupling constants h, h' . It is amusing to note 
that dct$ is a rcnormalizablc interaction only for the SU(3) xSU(3) case. 
One can easily verify that this superpotential allows a desired minimum 
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where 



(6) 



(7) 



A color-octet supermultiplet is absorbed into the heavy vector multiplet, 
leaving two singlet chiral supermultiplets and one color-octet chiral super- 
multiplet. Hereafter, we will only refer explicitly to the supersymmetric 
generalization of the model when the superparticle content has a significant 
impact on the model's phenomenology. 

Given the gauge symmetry of the full theory it is clear that we can only 
generate diagonal entries in the quark mass matrices. In order to generate 
realistic Cabibbo-Kobayashi-Maskawa (CKM) mixings, however, we require 
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off-diagonal entries in the basis of tlie interaction eigenstates as well. The 
way we can generate these off-diagonal components is through the exchange 
of heavy vector-like quarks @]. We introduce a minimal set of vector-like 
quarks, U, C and D, which have the same quantum numbers as the ordinary 
right-handed quarks ur, cr, and d^. Without a loss of generality, we can 
choose a basis where the invariant Dirac mass terms exist only for the U, C, 
and D fields and in which there is no mass mixing terms between U and ur, 
etc. Given the Higgs representations described earlier, the most general set 
of Yukawa couplings involving the vector-like quarks is given by 



+QiHD + QiHU + Q2HC + h.c, 



(8) 



where H is the Higgs doublet of the minimal Standard Model, and H = 
ia2H*. It is straightforward to identify which $ has to be replaced by $'* in 
the super symmetric case; H and H must be replaced by Hi and H2 of the 
Minimal Supersymmetric Standard Model (MSSM) as well. After integrating 
out the vector-like quarks and replacing the Higgs fields $1^2, '^'2,3 and H by 
their respective vacuum expectation values, we obtain the following effective 
mass terms 



V t'n J 

where the mass matrices have the form 

/ * * \ 



V b'n J 



+ h.c. 



(9) 
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Here each asterisk indicates a nonvanishing entry. By an appropriate choice 
of the vector-like quark masses and Yukawa couplings, we can obtain the 
following hierarchical form of the quark mass matrices without any fine- 
tuning of parameters 



/ rriu 
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/ rrid rrisX \ 

m, 
V / 



(12) 



where we have written our result in terms of the Wolfenstein parameterization 
of the CKM matrix. The (2,1) entry in M„ does not give us a physically 
significant effect since it can be eliminated by a rotation on ur and cr below 
the scale Ah- Note that all masses and mixing parameters of the minimal 
Standard Model appear in the mass matrices above, and it is straightforward 
to check that they reproduce the correct form of the CKM matrix.]^ In fact, 
we can even drop the (2,1) entry completely, and still reproduce the correct 
CKM matrix. 

One of our primary interests, however, is to consider the effects of our 
model on S-physics. Even though the mass matrices that we have con- 
structed can perfectly reproduce the CKM matrix, they do not at the moment 
have any effect on 5-physics because the first two generations are decoupled 
from the third in Ma] when we go to the mass eigenstate basis, we do not 
generate any new AB = 1 operators via Gh exchange. Therefore, we intro- 
duce an additional vector-like quark B that has the same quantum numbers 
as bfi. The new Yukawa couplings involving the B field are given by 



AC = 4$2,35 + QsHB. 



(13) 



After integrating out the B field, we obtain a single new entry in the down- 
quark mass matrix, which we assume is of the order m^A or less. Thus, 



/ rrid 


V 



nisX 



\ 

rris 
nibX^ nib J 



(14) 



where |^| ^ is a new complex parameter. 

■fThis rotation leads to D^-D^ mixing by the exchange of yet another color-octet vector 
boson coming from the breaking SU{i)i x SU{2>)2 S'f(3)i,2- However, we assume that 
this breaking scale is very high > 0(100) TeV to evade constraints from flavor-changing 
neutral currents processes. 

^Therefore, the form of the mass matrix is not a predictive ansatz despite an interesting 
texture. 
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It is important to note that the mass matrices we have constructed not 
only reproduce the correct CKM matrix (despite their many zeros) but they 
also lead to relatively small FCNC effects from Gh exchange, as we will see 
in the next two sections. In addition, our form of the mass matrices have the 
nice feature that Cabbibo mixing is generated in the down-sector while Vcb is 
generated in the up-sector. This is preferred from the point of view of model 
building, since it leaves open the possibility of explaining the famous relation 
A ~ ^Jmd/ms, and perhaps also Vcb — sjrnjmt. It is an interesting direction 
for future research to see whether it is possible to construct a realistic and 
predictive model within this framework, though we do not pursue this issue 
further here. 



3 Parameter Space 

In Figure 1 we show the allowed region of the M-cot 9 plane. The s-channel 
Gh exchange diagram can contribute significantly to the tt production cross 
section. We have adopted the central value of the tt cross section expected in 
the Standard Model from the next-to-leading order calculation by Laenen et 
a/.0, and have computed the shift of the production cross section in the qq 
channel expected in our model, including interference effects. The partonic 
cross section in the qq channel is given by 



a{qq tt) 




2mf 
1 + — 



R (15) 



where s is the parton center of mass energy squared, and R is given by 

M^(M^ + r^) 

(S - M2)2 + M2r2 ^ ^ 

The Gh width F is given by asM{A tan2 6 + cot2 6) j 6, for decay to 5 light fla- 
vors, for example. We use the MRS D- structure functions in computing the 
total qq cross section, to be consistent with Ref , and have assumed a K- 
factor of 1.2 to take into account next-to- leading-order effects. p| We require 
the total cross section to be within the 95% confidence level bounds of the 



^In fact, Laenen et al. use MRS D-' structure functions, but the difference is negligible 
for tt production. 
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value observed by CDF, a{ti) = G.StH pb, for = 176 ± 13 GeV|ig. Be- 
cause of the small statistics, the experimental errors are not exactly gaussian 
and there is a correlation between the errors in a{ti) and Unfortunately, 
the CDF paper does not show the variation of over an extended range 
of rrit, nor the dependence of the efficiency on m^. Thus, we have no way 
to determine the 95% confidence level bound rigorously. We simply double 
the error bar on the CDF cross section to obtain an allowed range, 2-14 
pb, for rrit between 150-202 GeV. Also, we do not take the correlation of 
the errors into account, and vary both a{tt) and rrit within the above ranges 
independent ly.|| We assumed a theoretical uncertainty in the central value 
of the Standard Model ti cross section that follows from varying the renor- 
malization scale between rrit/ 2 and 2mt. The excluded region lies within the 
oval area between ~400 and ~800 GeV. 

The dashed line in Figure 1 shows the bounds on new particles decaying 
to dijets at the 95% confidence level, from both UAl [|ll[] (between 200 and 
300 GeV) and CDF ]12[ (between 200 and 850 GeV). The Gh decays to 
quark-antiquark pairs, like the more familiar axigluon, but its coupling to 
light quarks is suppressed by a relative factor of 1/ cot 6. Thus, given the 
published bounds on the axigluon production cross section, we can determine 
the value of cot 9 necessary to suppress the Gh cross section until it is below 
the experimental bound. From this, we conclude that the region below the 
dashed line is excluded. Note that the older UAl bounds are more stringent 
below ~250 GeV due to higher statistics at smaller values of s. 

We also show the region allowed by the Z bb width measured at LEP. 
The new contribution to the parameter Rb = T{Z ^ bb) /T{Z hadrons) is 
given by 

ARb ^ |^a,(cot^ e - tan^ g)(l - Rb)F{M, Mz) 
Rb l + ^as[tan'^e + {cot^e-t8in'^e)Rb\F{M,Mz) ^ ' 

where the function F is provided in Appendix A. Note that there is another 
contribution in the supersymmetric version of the model which involves the 

"Even though this "rectangular" treatment of the errors correspond to (95%)^ = 90 % 
confidence level, the final bound is determined by the corners of the rectangle, and hence 
correspond to 98 % confidence level in the (uncorrelated) Gaussian case. The true confi- 
dence level depends on both the correlation and non-Gaussian nature of the errors. There- 
fore, the excluded region in our figure should only be thought of as an approximate bound. 
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superpartners of the G/z-boson and the bottom quark. However, this diagram 
is roughly one order of magnitude smaller than the result in ([T7|) [0 Again 
we require that the Standard Model value of Rb plus the contribution given 
above remain within the 95% confidence level bound of the value measured 
at LEP, 0.2210±0.0029 0. The dotdashed line indicates the experimental 
central value, while the solid lines put an upper and lower bound on cot 9. 
Since there is a small discrepancy between the measured value from the 
Standard Model prediction, the central value lies away from the line cot 6 = 1 
where the effect vanishes. f]\ Note that the analogous quantity defined for 
charm quarks, Rc, is reduced in this model if cot 6* > 1: 

ARc _ _ Rb ARb ^^g^ 
Rc 1 — Rb Rb 
Thus, when we account for the central value of Rb observed at LEP, we also 
obtain an —0.7% shift in R^ which is completely negligible compared to the 
experimental error, Rc = 0.171 ± 0.020 ||T|. Other Z-pole observables do 
not give us any further constraints. For instance, the forward-backward and 
polarization asymmetries in Z ^ bb are unaffected in our model because the 
Gh couples equally to left- and right-handed quarks. 

Finally, we show the region excluded by flavor changing neutral current 
bounds in the case where the parameter ^ = 0. The case of nonvanishing ^ 
will be discussed together with the associated B physics in the next section. 
Given the minimal {C, = 0) form of the mass matrices described in Section 2, 
there is no constraint from the down-sector, and the largest flavor changing 
neutral current bound comes from the D^-^ mass splitting. The effect is 
given approximately by 

Atud = —as • 2n rB^ofo^oVQCD (19 

9 iW'^sm t'cos'^fc' 



**0f course there could be an effect from the ordinary particle content of the MSSM, 
but it is negligible unless both the top squark and the chargino masses are just beyond 
the LEP limits. 

''"''In evaluating (|l7|), we have taken as{mz) = 0.110, as determined by lattice QCD, 
rather than the LEP value. This is because the observed enhancement m Z bb is roughly 
in a one to one correspondence with the high value of as{mz) observed at LEP. However, 
it does not change our figure significantly if we choose as{mz) = 0.120 instead. If we had 
used the preliminary number Rb = 0.2204 ± 0.0020 presented at Moriond, which is 2.4 a 
off from the Standard Model prediction, the allowed range lies in a narrower band above 
cot 9 = 1 everywhere, while the curve for the central value does not shift significantly. 
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where fo = 208 ± 37 is the decay constant as determined from a lattice 



QCD calculation ||I4[, and //qcd ~ 0.58 is the QCD correction from running 
the four-quark operator down from Mz toruD, using the conventions of Buras 
et al. |T^. Q We take Br){mc) = 1 in our numerical analysis. We require that 
(|19D is less than the experimental upper bound Atjid < 1.32 x 10"^° MeV to 
determine the excluded region of the parameter space. This is appropriate 
because the Standard Model contribution to D^-D^ mixing is expected to 
be ~ 5 orders of magnitude smaller than the current experimental limit 
1^ . The excluded region lies to the left of the parabolic solid line shown in 



Figure 1. 

The exchange of the color-octet Higgs boson also contributes to flavor- 
changing processes, but with a much smaller effect. The ^^€'£^$2,3 coupling 
is of order rritAX'^ / {(^2,3) ^ and the coupling to the mass eigenstates crUl is 
suppressed by an additional factor of A\^{mc/mt) x \{p — irf). Thus, the 
contribution to Am^) from Higgs exchange is down by at least {nic/mtY 
compared to (19). We also checked that the exotic decay modes t ^ eg, cy 
and ebb are negligible compared to the Standard Model decay t — > bW. 

Taking all the constraints into account, we see from Figure 1 that there are 
two allowed "windows" , in our model's parameter space: one small region at 
approximately M ^ 250 GeV, and a somewhat larger region above M ^ 600 
GeV. If we further require that the central experimental value of Rf, from 
LEP has to be reproduced, we must take cot 6* ~ 2 in the light window and 
cot ^ ~ 4 in the heavy window. 



4 5-physics 

Using the ^ = form of the down-quark mass matrix, we showed in the 
previous section that we could account for Rf, within the two allowed regions 
of our model's parameter space. In this section, we will consider the effects of 
the nonminimal 7^ 0) form of on 5-physics. The question we would like 
to address is whether we can also reduce the 5-meson semileptonic branching 
ratio BsL and the charm multiplicity per decay Uc through the additional 

ttWe consistently ignore the effects of running between fi = M and mz, since 
log{rn,z /rri},) ^ \og{M/mz) and as(/i > mz) < asi^i < mz) as well. 
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AB = 1 operators that are present when ^ 7^ 0, without conflicting with the 
constraints from FCNC processes. We will argue that our results strongly 
favor the supersymmetric version of the model. 

With ^ 7^ 0, we generate the following effective AB = 1 operator through 
the exchange of the Gh boson. 

^-ff = ^^^^R^'T-^r{u^^^T-u + d^^T'^d + s^T'^s + c^^T-c), (20) 

M^COS"^t^ 

Since this operator always has an sr in the final state, the amplitude does 
not interfere with the Standard Model amplitude (which only involves sl) 
providing that we neglect compared to mb- Therefore, we add the new 
decay amplitude incoherently to that of the Standard Model. The QCD 
running modifies the normalization of the operator from the weak scale to 
the scale m;,; we take this into account in our numerical analysis below. 

We estimate the additional contributions to the B partial decay widths 
from the effective operator in ( PO] ) using the parton-level spectator quark 
approximation. We find 

where the function G is defined by 



G{y) = ^yr^(16-40y + 18y2-V) + ^(8-3i/)y='lnl±^J, (22) 

and is normalized such that G{0) = 1. 

As we described earlier, the current experimental data suggests that both 
the B semileptonic branching ratio Bsl, and the charm multiplicity per B 
decay Uc, are lower than the Standard Model expectations. Notice that 
decays via the operator ( pOf ) increase the total decay rate without affecting 
the semileptonic mode, and hence Bsl decreases. This operator also reduces 
ric since it contributes to the sqq final state, which involves charm quarks 
only part of the time. We list the effects on Bsl and ric for various choices 
of the parameters in Table. 1. The effect of Gh exchange is described by the 
parameter A, defined by 

, /750 GeV\^ 
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masses 


heavy 


experiment 


asimz) 


0.105 0.115 




BsL 


15.4(1 - 0.22A) 12.7(1 - 0.28A) 


10.6 ±0.3 18 


ric 


1.09(1 - 0.21A) 1.09(1 - 0.25A) 


1.08 ±.06 



Table 1: Effect of Gh exchange on S-decay for the choice of "heavy" masses, 
rric = 1.7 GeV and = 5.0 GeV. The renormahzation scale of the Standard 
Model operator is chosen at /i = m^. 



masses 


light 


experiment 


asimz) 


0.105 0.115 




BsL 


13.2( 


1-0.16A) 12.7(1 - 0.20A) 


10.6 ±0.3 


18 




Tic 


1.21( 


1-0.13A) 1.21(1 - 0.16A) 


1.08 ±.06 


20 





Table 2: Effect of Gh exchange on 5-decay for the choice of "light" masses, 
rric = 1.2 GeV and mb = 4.6 GeV. The renormahzation scale of the Standard 
Model operator is chosen at fi = rrib. 



We used the Standard Model predictions quoted in Ref. |]T8[ with the renor- 
mahzation scale /i = nif,. For /i = mf,/2, Bsl decrease by only 4%, while 
ric does not change at all. The "heavy" case corresponds to = 1.7 GeV, 
rub = 5.0 GeV, and the "light" case to = 1.2 GeV, rub = 4.6 GeV. QCD 
renormahzation effects were taken into account by numerically solving the 
renormahzation group equation using the anomalous dimension matrix given 
in Ref. . For the "heavy " quark masses it is difficult to reproduce the ex- 
perimental values for any choice of A. However, one must keep in mind that 
the value of Bsl corresponding to the "heavy" quark masses is very far from 
the experimental data for A = 0, indicating a more serious problem from 
the beginning. For "light" quark masses the discrepancy with the Standard 
Model predictions are ~ 20% for both Bsl and ric, and we can easily obtain 
these shifts by an appropriate choice of the mixing parameter ^, providing we 
do not exceed the bounds on other flavor changing processes, as we describe 
below. 

Now that we have introduced mixing between Br and sr, we must also 
consider the constraints from the other FCNC processes associated with a 
nonvanishing value of C,- We found that the most stringent constraint comes 
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from the contribution to B^-^ mixing from the operator 

^=1 (m-^a ^^'f ) ' bRYnnbn^.nn, (24) 

2 \M smt/coso' / 

Here, [/'^^ is the rotation matrix acting on the right-handed down-type 
quarks, and U^i ~ X^{Xmd/ms) ~ A''^ is the mixing angle between and 
(Ir. The renormahzation of this operator down to the scale rrib gives us the 
correction factor tiqcd — 0.55, again using the definition of r/gcD given in 



Ref [15|. The contribution to the mass splitting is then given by 




4c ^ 2 



A-B = y ( Lose 1 fBBml,^cn , (25) 



where Bf'^ = (1.0 ± 0.2) (180 ± 50 MeV)^ , based on recent lattice estimates 
[|T^. As before, we obtain much smaller effects from octet Higgs exchange. 



The sum of the Standard Model contribution AmBlgpf and AtubIq^ shown 
above should give us the experimental value 

^'^bIsm + ^^b\gh \ = ^^Blxpt (26) 

where Amslg^^^ = 3.357 x lO^^'^ GeV. However, the Standard Model predic- 
tion depends sensitively on the choice of Vtd- If we fix = 176 GeV and 
Bf^ = (180 MeV)^, then Am^l^Aj can range between (0.16-2.50) Arusl^^p^ 
[ p^ if we allow Vtd to vary within the 90% CL. range allowed by CKM uni- 
tarity Vtd = 0.004-0.015 [||]. Since the phase of the parameter ^ is arbitrary. 



we obtain the bound 



Am 



b\g, 



< 3.5 Amsl^^pt, hence 



3.11 TeV 1 

M < = . (27) 

VA cot^ ^ ^ 

Since we have used the 90% CL. upper bound on Vtd in deriving this expres- 
sion while keeping rrit and -B/^ fixed at their central values, it follows that 
the actual 95% CL. bound is weaker than the one given in (^Tf ). We use this 
strict bound to demonstrate that even with conservative assumptions there 
exists an allowed range in the parameter space of the model when A ^ 1. If 
we choose A = 1, so that we can account for a 20% reduction in Bsl and 
He, then we must also be below the dashed line shown in Figure 2 to evade 



the B-B mixing constraint given by (27). Thus far, we see that both in the 
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light and heavy windows, we can explain the anomalies in Bsl, "^c, and Rb 
simultaneously. 

The only other potentially significant FCNC constraint that depends on 
the parameter ^ and that may alter our conclusions is the 6 ^ 57 branching 
fraction. Note that all of the flavor-changing processes that we have consid- 
ered until now have received contributions at tree-level in our model, while 
the b —>■ s'-f width receives contributions at the one-loop level only. This point 
is significant because the supersymmetric particle content of our model can 
now have a dramatic effect on the result. 

Let us consider the upper bound that we can place on cot 6 from the b — 
57 branching fraction. The most important contributions to the amplitude 
for cot^^ > 1 come from the Feynman diagrams proportional to 1/ sin^ ^, 
shown in Figure 3. If we first consider the non-supersymmetric case, the 
entire nonstandard contribution to the inclusive branching fraction comes 
from the first diagram. If we compare this to the difference between the 
CLEO upper bound, 4.2 x 10^"^ |22| and the Standard Model expectation, 
1.9 ± 0.54 ||23|, and require agreement within two standard deviations, then 
we obtain the constraint 

cot0<1.7/v^ (28) 

If we require A = 1 to explain the -B-decay anomalies, then we see that (p^) 
forces us to live in the lower right handed corner of the allowed region in 
Figure 2, where we cannot simultaneously explain the central value of Rb- 

However, the situation changes dramatically in the supersymmetric case. 
It is well known that the b s'j branching fraction vanishes identically in 
the limit of exact supersymmetry |^ . Thus, the new contribution to 6 ^ S7 
in our model not only depends on M and cot 9, but also on the soft-SUSY 
breaking masses and couplings. In addition to the diagram involving ordinary 
particles, there are a number of other diagrams that involve the exchange of 
the left- and right-handed 6-squarks, [bi, bn) and the fermionic partners of 
the transverse and longitudinal components of the G/f-boson [ip, x)- The new 
diagrams tend to cancel the first one. An exact result for these diagrams is 
provided in Appendix B. If we assume that the soft SUSY-breaking masses 
defined in Appendix B are around 100 GeV, temporarily ignore 6l-6_r mixing 
effects (i.e. ignore the fourth diagram), and compare the total result for the 
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inclusive branching fraction to the CLEO upper bound, we now find 

cot^ < 3.2/v^ (M = 250 GeV), 

cot^ < 5.5/v^ (M = 640 GeV). (29) 

correspoding to bound (a) in Figure 2. In this case we do not exclude the 
interesting regions of our parameter space when A = 1. We show the regions 
excluded for A = 1 and for various choices of the SUSY breaking parameters 
in Figure 2. Note that we have not included the QCD running effects in 
these bounds since the relevant anomalous dimension matrix is not available 
in the literature. While this effect may give an important correction to our 
estimates above, our conclusions will not change once we take into account 
the strong dependence of our results on the soft supersymmetry-breaking 
parameters. For example the running of the 6 — S7 operator itself results 
in a suppression in the amplitude by a factor of ~ 0.7, while the four-fermi 
operator Eq. (^) may feed into the 6 — > 57 operator at the two-loop level 
and enhance the amplitude by a factor of 2 or 3. However, if we include 
the bi-bji mixing in the fourth diagram, it is easy to obtain a suppression of 
approximately the same size without any real fine-tuning {e.g. the difference 
between bounds (b) and (c) in Figure 2). This is possible because we do not 
know the magnitude or phase of the soft SUSY-breaking parameter A. With 
A ^ 100 GeV, the bounds in (p9D become 

cot^ < 3.8/v^ (M = 250 GeV), 

cot^ < 8.6/v^ (M = 640 GeV). (30) 

While a complete study of this enlarged parameter space is beyond the scope 
of this paper, it is clear that the suppression of 6 — > S7 in the supersymmetric 
case, as well as the additional parametric degrees of freedom prevent us from 
excluding additional regions of the M-cot 6* plane when A = 1 (c./. bound 
(d) in Figure 2). Thus, we can account for the observed value of Rb, and 
obtain a 20% reduction in Bsl and ric simultaneously. 

5 Confinement Phase of SU(3)3 

We have seen that it is possible to explain the anomalies in Rb, Bsl and 
ric in both allowed regions of our model's parameter space. However, the 
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strong possibility that the hghter window (M ^ 250 GeV) may be ruled out 
in the near future by the improving bounds on new particles decaying to 
dijets makes the heavier region (M ^ 600 GeV) worthy of further consid- 
eration. In order to reproduce the central value of Rb in this region would 
require that we take cot6' ~ 4, which implies a rather large SU{3)3 gauge 
coupling. This leaves open the possibility of nontrivial nonperturbative dy- 
namics. For example, if the dynamics of the third family in the limit of 
large is like the Nambu-Jona-Lasinio model, then we would expect chiral 



symmetry breaking at a critical coupling of cot 6* = yTT/{3as) ~ 3.4, and a 
large dynamical mass for both the top and bottom quarks. This would not 
be phenomenologically acceptable. However, what actually happens in this 
limit depends on the nonperturbative dynamics and on the mass of the Higgs 
boson $2,3- In this section, we consider the possibility that the SU(3)3 gauge 
theory may be confining, and that the physical bottom and top quarks may 
be composite particles, bound states of the fundamental quark fields and the 
Higgs field $2,3- We will argue that the phenomenology of such a confin- 
ing phase is the same as the Higgs phase we have described in the previous 
three sections, providing that the third family dynamics is similar to the 
Abbott-Farhi model g. 

Let us consider what happens ii ^ 1 and SU(3)3 is confining. If $2,3 
is much heavier than the scale at which g^ becomes large, then $2,3 decou- 
ples from the dynamics and we expect spontaneous chiral symmetry break- 
down, and a large dynamical mass for the third generation quarks. If $2,3 
is lighter than the scale at which the SU (3)3 coupling becomes strong, then 
complementarity [^| suggests that the resulting confinement phase should 
be smoothly connected to the Higgs phase described earlier in this paper. 
Thus, we expect the physical top and bottom quarks to become composites 
of the fundamental top and bottom quarks with the $2,3 boson. We identify 
physical composite top and bottom states, tc and be, with the composite op- 
erators ($t)/A and ($6)/A, where A is the compositeness scale. Note that 
these composite states transform as triplets under SU{3)i^2 as desired. In 
this phase it is not necessary to have (it) or (bb) condensates since the low- 
energy fermion content satisfies the 'tHooft anomaly matching conditions, as 
in the Abbott-Farhi model 1^. There is a p-like meson state in the confining 
phase composed of tt, bb and $$* bound states, which corresponds to the Gh 



16 



boson of the Higgs phase. The composite top and bottom quarks would re- 
main massless if it were not for the exphcit chiral symmetry breaking effects 
of their associated Yukawa couphngs, and hence the masses of the composite 
states are proportional to their Yukawa couplings as before. 

All the phenomenological constraints that we have discussed also carry 
over smoothly to the confining picture. For example, let us consider the qq 
tt cross section presented in Section ^ The only contribution to this process 
in the confining picture is s-channel gluon exchange, but now there is a form 
factor F{q^) at the gluon-tc^c vertex. Assuming vector-meson dominance, 
F{q^) is saturated by the exchange of the p-like meson {Gh), which mixes 
with the gluon. Since F{q'^) should reduce to the form factor for a point-like 
interaction F{q^) — 1 in the ^ limit, it must be of the form 



for small q^. Thus, the cross section computed in the confining phase has 
exactly the same dependence on M and F as the cross section we obtained 
from perturbative Gh exchange in the Higgs phase. We can also describe the 
phenomenology of the confining phase in terms of the same parameter space 
as the Higgs phase, providing that q^ <^ M^. The p-like meson does not cou- 
ple to the light quarks directly, while the coupling to and he is an unknown 
parameter that we can identify with Qs cot 9. Therefore, the correction to i?b 
involves the same combination of parameters as before. The dimension-five 
operators generated by the exchange of vector-like fermions induce mixing 
between elementary light quarks and composite third generation quarks, and 
the flavor off-diagonal coupling of the p-like meson is proportional to Qs cot 9, 
again consistent with the Higgs phase result in the limit cot 6' — > oo. 

6 Conclusions 

We have shown that the extension of the Standard Model presented in this 
paper can yield corrections to Rb-, the 5-meson semileptonic branching ratio 
and the charm multiplicity in S-decays, in a pattern roughly consistent with 
the experimental data. In the nonsupersymmetric version of the model, we 
can explain either the value of or the 5-decay anomalies within the al- 




-M2 + iMV 



(31) 
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lowed parameter space of our model, but not both simultaneously. This is 
a consequence of the strong upper bound on cot 6 that we obtain from the 
limit on 6 — i> 57, when we take A = 1. However, the natural suppression 
of 6 — 57 in the supersymmetric version of the model allows for choices of 
the parameters that can account for i?^ and the -B-decay anomalies simulta- 
neously. Because the massive color octet bosons in this model couple more 
strongly to third generation quarks, a natural way to test this idea is to search 
for a high mass peak in the hb dijet invariant mass distribution at hadron 
colliders. This may rule out the small allowed window that we found near 
M 250 GeV in the near future. [| At present it is interesting to note that 



the most recent search at CDF |]T2[ for new particles decaying to dijets has 



found upward statistical fluctuations in the data near 250, 550, and 850 GeV 



1^. While probably nothing more than coincidental, it is amusing to note 
that two of these masses fall within the two allowed regions in our model's 
parameter space. Finally, we have considered the limit in which the SU(3) 
associated with the third family quarks becomes confining, and argued that 
the bottom and top quarks may become composite objects without forming 
condensates, like the composite fermions of the Abbott-Farhi Model. We ar- 
gued that the phenomenology of the confinement phase should be smoothly 
connected to that of the Higgs phase described in this paper. Thus, it is 
possible that third family compositeness effects may give us other nontrivial 
signatures of this model. 
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A Appendix 



The function F{M, Mz) defined in Eq. 17 is given by 



F(M, Mz) = F^ + F2 (32) 



*A search is currently under way at CDF, but the data analysis has not yet been 
completed. 
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for M < mz and 

F(M, Mz) = F2 

for M > mz, where 

Fi = (1 + (5)2 [3 In 5+ (In 5)^1 +5(1-52 
-2(1 + 5)2 



2(5 In 5 



ln(l + (5) In (5 + Li2 (y^) - Li2 [ y ^ 



+ () 



F2 = -2|^ + (5 + ((5+^)ln(5 
+(l + (5)2 



l + (5/ 2 U + (5/ 6 



(33) 



(34) 



(35) 



Here Li2{x) = — Jq y ^^(^ ~ ^) Spence function, and 5 = M"^ /m\. 



B Appendix 

The effective operator yielding the largest contribution to to 6 — > S7 for 
cot ^ > 1 is given by 



36;^^M^4-2"^^ ^"^^^ 



(36) 



where F^p is the photon field strength tensor, and the index i runs over the 
four classes of diagrams shown in Figure 3. We find to lowest order in the 
bottom quark mass: 



Cl 

C3 
C4 



4 
3 



-E 
-E 



U^jU^jM 



l-r,^ + 2r2ln(r|) 



(1 - r2)3 



= 2A^ 



-^2 



{i-pir 

-4 + (l+p2)(l + ^2) 



(37) 
(38) 

(39) 



(l-r2)3(rf-p|) 



2(l-p2)2(l_^2)2 

Inr?- 



(l-pf)^(pf-r2) 



lnp2 



(40) 
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where we have neglected terms of order ml/M'^. Above the fii are the eigen- 
values of the ip-x mass matrix 



M 

M 



(41) 



where ip is the superpartner of the the transverse component of the Gh-, and 
X is the superpartner of the would-be Nambu Goldstone boson in $2,3- Note 
that the off-diagonal entries are generated by the vacuum expectation value 
of $2,3; while the diagonal ones are SUSY breaking effects. U is the matrix 
that diagonalizes = 771^^/ fii, and pi = mi^/fii. In the final diagram, 

Arrih parameterizes the 6l-&_r mass squared mixing, where A is the trilinear 
soft SUSY breaking parameter. 
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Figure Captions 



Fig. 1. Allowed regions of the M-cot^ plane, for ^ = 0. The region 
above (below) the upper (lower) solid line is excluded by Rb, the region below 
the dashed line is excluded by searches at UAl and CDF for new particles 
decaying to dijets, the region inside the dashed "oval" is excluded by the ti 
production cross section, and the region to the left of the sohd parabola is 
excluded by D-D mixing. The dotdashed line shows the central value for Rf, 
measured at LEP. 

Fig. 2. Allowed regions of the M-cot^ plane, for excluded by B-B 
mixing, for the choice of A shown. The region above each dashed hne is 
excluded by the 6 — > 57 constraint for A = 1, and for various choices 
of the soft SUSY breaking masses {rni^, m^^, m^, m^. A). The dashed 
Unes correspond to the following mass sets (in GcV): (a) (100,100,100,100,0), 
(b) (200,200,200,200,0), (c) (200,200,200,200,200), (d) (300,100,200,200,200). 
The non-supersymmetric result is also shown. 

Fig. 3. Feynman diagrams proportional to 1/ sin^ that contribute to 
6 ^ 57. The numbering of the diagrams corresponds to the results presented 
in Appendix B. 
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Fig. 3 




cos 6 sin 6 



